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1. Introduction 


In the last years, numerical simulation has seen a great development thanks to costs 
reduction and speed increases of the computational systems. With these improvements, the 
mathematical algorithms are able to work properly with more realistic problems. 
Nowadays, the solution of a problem using numerical simulation is not just finding a result, 
but also to ensure the quality. However, can we say that the model results are correct 
regarding the behaviour of the system? In other words, how could we quantify the 
similarity between reality and simulations? To answer these questions, it is necessary to 
establish a validation criterion that allows an objective quantification of the difference 
between the results and the reality. Another way to say this is, how “true” our results are. 
In the case of numerical methods, the main objective is to replicate as closely as possible the 
behaviour of the "real" world through numbers. Normally, the results of the numerical 
methods are expressed in terms of graphics, pictures, etc. These results represent the view of 
reality that the chosen method provides (Oñate, 1998). In order to affirm that the result of a 
numerical solution is fully consistent with the reality, it must be satisfied that: 
a. The mathematical model must incorporate all aspects of the real world. 
b. The numerical method has to solve exactly the equations of the mathematical 
modelling. 
The problem starts with these two conditions that guarantee the "truth" of the results, since 
none of them are fully accomplished and it must be admitted that the numerical prediction 
never completely matches the "real" world behaviour. Then you can only be sure that the 
numerical solution is a good approximation of the reality. Now, new questions arise: How 
much does the result obtained by a numerical method resemble the reality? How can we 
objectively quantify this similarity? The answers to these questions are those that give rise to 
the validation methods. 


2. Types of validation 


All validation is done through a comparison of a pattern or a reference model with the 

model under study. There are many ways to make a validation, but in general they are 

usually classified according to the pattern used in the comparison (Godoy & Dardati, 2001), 

(Archambeault & Connor, 2008): 

a. Validation using other numerical solutions. This technique compares the results to be 
validated with the results obtained through other numerical methods previously 
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validated. In other words, one technique has been validated before it can be used as a 
reference to validate the second method. 

Another way to use this technique is using more than one numerical method to solve 
the problem. If the physics of the problem is properly modelled in all the techniques 
used, the results should have a clear trend and similarity; therefore, knowing the 
advantages and disadvantages of each technique it will be possible to perform a 
validation of our technique. 

b. Validation using analytical solutions. This type of comparison can be used when the 
researcher knows the analytical theory behind the problem and makes a direct 
comparison of the simulation results with the analytical solution. One of the main 
problems of this technique is that it can only be used in extremely simple cases, 
because trying to find the analytical solution of real problems is almost impossible (in 
fact, this is the reason of making numerical simulations). However, this technique is 
useful when you want to validate the code of the numerical method. Through the 
analytical solution it is possible to obtain the exact value of the problem, thereby 
reducing the external variables that can affect the results. 

c. Validation using experimental results. This technique is the most popular of all; this is 
due mainly to the fact that the measurement shows the consistency of the model with 
the reality. However, one cannot forget that whenever you perform a measurement you 
should introduce a measuring instrument and this directly or indirectly affect the 
system being measured (Archambeault & Connor, 2008). For this reason, it is essential 
to have the greatest similarity between the measurement and simulation configurations. 
For the real environment (measurement) one should take into account the possible 
limitations of the laboratory and the equipment required to perform the measurement. 
The most important issue is to narrow down to a minimum any device that cannot be 
fully simulated such as cables, connectors, etc. On the other hand, for the computing 
environment (simulation) one should try to model the entire possible setup or at least, 
include the most important characteristics. Otherwise, it runs the risk that the 
simulation results do not represent the reality faithfully, causing a validation error. 

d. Validation using intermediate results. This technique compares the intermediate 
results of the numerical model with experimental or theoretical known values, although 
these results are not the final objective of our comparison. The major drawback of this 
method is to find an intermediate result that is really close related with the final result 
under study. On the other hand, it is very easy to lose sight of the factors that could 
affect the intermediate variable making the comparison of the final result not valid. 
However, this technique is frequently used to monitor some parameters of the 
numerical simulations, but it is rarely used alone or as a main validation method. 

A good example of this technique can be found in electromagnetic simulations. Imagine 
that it is required to compare the far-field simulations and measurements produced by 
a source inside an airplane. In this case, to make far-field measurements in a structure 
as big as an aircraft can be very expensive and complicated. However, it is possible to 
measure some near-field values at specific points near the aircraft and to compare them 
with simulations results calculated at the same points. Based on the direct relationship 
between the far-field and near-field, the similitude between simulation and existing 
measurements in the near field will be proportionately the same than in the far-field. As 
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can be seen in this example, the validation is not done with the final results (far-field), 
but an intermediate result (near-field) is used to set a criterion of similarity between 
simulation and measurement. 
Validation using convergence. This type of validation is based on a comparison of the 
convergence of the numerical model with the pattern or the reference results. This 
comparison is done knowing that the solution found is not the best, but assuming that 
the model results converge. 
Another situation where this type of validation can be used is when it is impossible to 
get a "pattern" to do the comparison. A good example of this type of validation can be 
found in chaos in structural mechanics area (Awrejcewicz & Krysko, 2008), where the 
validation is used in two ways; first, using a method to solve the system individually 
and observing if it converges or not. The second one is to use various methods to 
analyze the system at the same time and consider whether they all converge towards 
the same result and which of them do it faster. 
In the electromagnetic area this technique of validation is often used when we want to 
know a general behaviour of the system in a very short time. Normally, a very simple 
model with very coarse meshes is simulated. Then, the most important resonances and 
the general behaviour are observed to analyze whether or not we are on track. The only 
drawback of this technique is that it is not recommended for use as a final method of 
validation, because convergence of the system cannot be guaranteed. 
Regardless of the type of comparison that is performed (analytical, numerical, experimental, 
etc) at the end, the validation process is reduced to the comparison of the results and, in 
many cases, to the comparison of a pair of graphs. After that, once the type of validation to 
be performed in our model is chosen, the problem is how we can compare our results with 
the pattern in a quantitative mode. In many fields of research, simple visual inspection is 
used as the validation method when faced with the need to compare their results with the 
established models or patterns. 
Although the visual comparison is used in different environments with apparent reliability, 
it has potential limitations. Among the most common problems are: 
e The eye concentrates on peak positions and ignores the poor correlation of 
intensities (D. E. Coleby & A. P. Duffy, 2002). 
e Due to the potentially subjective nature of the visual comparison, the results 
produced cannot be used with confidence (D. E. Coleby & A. P. Duffy, 2002),(A. 
Duffy, D. Coleby, A. Martin, M. Woolfson, & T. Benson, 2003). 
e Comparing and quantifying the results objectively between different groups of 
experts can be difficult (Williams, M. S. Woolfson, T. M. Benson, & A. P. Duffy, 
1997), (A. Duffy, D. Coleby, A. Martin, M. Woolfson, & T. Benson, 2003). 
e The data may be too large (either a high volume of data or a very complex 
topography) to be compared visually with ease (Williams et al., 1997). 
These limitations force the need to investigate reliable and objective computational 
techniques to compare the differences of the results and evaluate their quality. 


3. Validation methods 


Numerous studies show that a direct comparison point by point is not feasible when large 
amounts of data are compared (D. E. Coleby & A. P. Duffy, 2002), (Drozd, 2005), 
(Archambeault & Connor, 2008). Therefore, this method is not recommended to validate the 
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results and much less to assign an absolute value of accuracy. This approach makes sense 
only for simple models, but in the numerical simulations, the results are often very complex. 
Today there are several methods of validation. Among the most used are: 


3.1 Correlation 

This is a widely used method for its ease implementation and interpretation and it is 
intended for quantitative variables for which there is a linear relationship. The correlation 
between two variables is perfect when output value is closest to 1 or -1 and gets worse as it 
approaches to 0 (D. E. Coleby & A. P. Duffy, 2002). The sign indicates the direction of the 
association: a value equal +1 indicates a perfect positive linear relationship. When this case 
happens the relationship between two variables has exactly the same behaviour: when one 
of them increases, the other increases too. If instead of that, the value is -1, it is said that 
there is a perfect negative relationship and implies that both signals have a linear 
relationships, one will decrease as the other increases. 

The most popular type of correlation is called the "Pearson correlation coefficient" and is 
usually used to measure the strength of the relationship between two variables when there 
is a linear relationship between them. 

The Pearson correlation coefficient is defined by the following expression: 


(Yeo Y1o - LiLo Y2q) 
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pe (1) 
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Where Y1\) is the dataset 1, Y2() the dataset 2 and n is the total number of points in both data 
sets. 

The major limitation of this correlation technique is that it only can be used when the 
relationship between the variables is linear. This means that when variables are closely 
related, but not linearly, the validation results can not reflect the expert opinion. There are 
very few cases that have a linear relationship between variables so this method is only used 
for extremely basic cases. 

An additional problem with this method, even when the data sets to compare have a linear 
relationship, is the interpretation a determined coefficient value. Or how does one know if a 
value is high or low? The answers to these questions depend largely on the nature of the 
investigation and the sample size used. For example, a correlation of 0.01 may be significant 
in a sufficiently large sample and a 0.9 may not be in a small sample. The law of large 
numbers is fulfilled, being that the weak trends are very unlikely from a null hypothesis and 
large amounts of data, while strong trends may be relatively likely in the small data size. 


3.2 Reliability factor 

This method is known as the R-Factor (The Reliability Factor) and it is one of the main 
criteria accepted in the validation area. The R factor could be considered a type of 
correlation; it is an objective method that provides with a single number, the similarity 
between two data. This method was created mainly to compare the intensities between the 
experimental and theoretical results in structural determinations of X-rays. A variation of 
different R-factors have been proposed: The first of these was introduced by Zanazzi and 
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Jona (Zanazzi & Jona, 1977), followed by Van Hove (Van Hove, 1977) and Pendry (Pendry, 
1980). 


3.2.1 Zanazzi and Jona R-factor 

The R-factor of Zanazzi and Jona is also known as "Rz -factor" (Zanazzi & Jona, 1977), and 
was planned to study the similarity of X-Ray diffraction and surface crystallography. 
Further studies modified their original equations and applied them in the numerical 
simulations area (Williams et al., 1997). This method was designed to accentuate the 
maximum slopes rather than the heights. This is accomplished by comparing the gradients 
of the two signals you want to compare. It basically made the differences between the 
signals for the first and second derivatives, accentuating the features present; making the R- 
Factor sensitive to positional changes in the data. The equation used to calculate this R- 
Factor is as follows: 


p, = ico WG). FQ) (2) 
V1) 
wif) = [V1 — C.¥2G, 6) 
|Y 1ipl + |max(¥1())| 
F(f) = |¥lgy — €.¥2%| (4) 
_ dito Y1¢) (5) 
dito Y2 


Where Y1’ and Y1ẹ” is the first and the second derivative of dataset 1. Y2’ and Y2” is 
the first and the second derivative of dataset 2. Y1 is normally used with the experimental 
value Y2 obtained and the theoretical value or reference pattern. C is used to adjust the 
intensity in both dataset. 

This technique is useful when comparing sharp signals with peaks where the importance 
lies in their peaks or valleys, but does not offer the same reliability when you want to 
compare noisy signals where the peaks are not as marked and the variations between 
valleys is very fast. 


3.2.2 Pendry R-factor 

The Pendry R-Factor (Pendry, 1980) is used generally to measure the degree of correlation 
between two signals that have many variations in their maximum positions. This method 
uses derivatives in place of their intensities; this is attributed to all peaks of the same weight, 
regardless of the height of each one. The idea of this technique is that any maximum 
contains structural information due a constructive interference. Thus the maximum 
occurring at high energies are generally lower than those obtained at low. The equation 
used to calculate this R-factor is as follows: 
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Yorn = Eset (7) 
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Where L is the intensity and L’ is the differentiated intensity in each dataset. Y1¢ is the 
dataset 1, Y2 the dataset 2 and n is the total number of points in both data sets. 

Unlike “Zanazzi and Jona” factor where it is necessary to calculate the second derivative, 
the Pendry R-Factor only requires the first derivative, making it less susceptible to small or 
rapid changes. This feature makes it a useful tool for analyzing very noisy signals. However, 
the main problem with the Pendry method is that it requires finding an adjustable 
parameter that is not constant (Robertson et al., 1990), which seriously restricts the use of 
this technique. 


3.2.3 Van Hove R-factor 

The technique of Van Hove (Van Hove, 1977) is the most widespread of all the R-Factor 
ones. This technique uses five different equations (9)-(15) to compare the position and width 
of the signal peaks; the shape of the peaks and the troughs, their number and their heights 
(D. E. Coleby & A. P. Duffy, 2002). The different indicators of this method are calculated 
using the following equations: 


Eol Y 1o — C. Y2 


R = (9) 
Xi-olY1ol 
_ Dizol 1a) (10) 
Èi- Y20 
2 
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Where Y1œ is the dataset 1 and Y2,) is the dataset 2. Both indicators (R1&R2) show the 
similarity in positions, heights and widths of peaks and troughs. 


_ N°slopes*(Y1) N° slopes*(Y2) (12) 
3 N? slopes-(Y1) N? slopes-(Y2) 


Where N’slopes+ is the number of positive slopes and N’°slopes* is the number of negative 
slopes for each dataset. The R3 indicator compares the number of positive slopes with the 
negative slopes of the opposite graph. 


= - - (13) 
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Where YT is the first derivate of dataset 1 and Y? is the first derivate of dataset 2. In this 
case the R4 and R5 indicators are used to compare the gradient of the data sets. 
Finally, the Van Hove Factor has a very useful indicator that combines all indicators to 


calculate the total difference between the two graphs, it is called: "Rr" (15). This indicator 
allows to quickly and accurately getting an overall idea of how good a result is with regard 


to the pattern. 
Rr = [R2 + R2 + R2 + R2 + R2 (15) 


3.3 Integrated with logarithmic frequency error (IELF) 

The idea behind IELF (Integrated against Error Log Frequency) is based on the premise that 
when comparing data with a very high feature density, the overriding factor to be assessed 
is a function of the difference between the two traces (Simpson, Jones, MacDiarmid, A. 
Duffy, & D. Coleby, 2005). Basically, this method is the difference between two traces in 
logarithmic axis and in frequency domain. Then the result is integrated (summing) to get a 
single value. The IELF equation is given in (16): 


l i 
Xg lerror;l. [ng (i+) 7 sew) 
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Where f are the frequency points being compared (from point 0 to point n, resulting in n+1 
discrete frequencies), |error| is the difference between the two data sets at the nth data 
point. 

There is an improved version of IELF method known as IELF modified (IELFyop). This 
modification involves summing the elements halfway between the data points in order to 
improve the approximation of the difference in the measured data. This modification is 
given in equation (17): 


ore, Infern tf InGi-y + fi-1) 
ane ki [errorg)|.| ——3 —— - r N (17) 
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Today, the ELFmop method is widely used to validate large volumes of data. This method is 
used in circumstances in which the data to be compared has a high visual density, ie, it is 
impossible to differentiate visually (Knockaert, Catrysse, & Belmans, 2006), (A. Duffy, D. 
Coleby, A. Martin, M. Woolfson, & T. Benson, 2003). The main disadvantage of this method 
is that it has very few tools to perform a good validation; only one indicator is very little to 
interpret all the aspects present in the validation process. Another important disadvantage 
is that it is defined only for the frequency domain in logarithmic axis and there are some 
weaknesses with abrupt changes in graphics. 
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3.4 Feature Selective Validation Method (FSV) 

The method of Feature Selective Validation (FSV) was developed by Anthony Martin and 

Alistair Duffy in 1999 (A. Martin, 1999) and today is the method most widely used because 

of its versatility and simplicity. This method is widespread and is currently being developed 

as a standard of validation for Computational ElectroMagnetics (CEM) within the project by 

IEEE 1597.1/1597.2 (Standard IEEE, 2008). The Feature Selective Validation (FSV) method 

was developed with the specific aim to reflect the approach taken by engineers when 

assessing data presented visually during the validation of computational electromagnetic 
simulation. Today it is possible to find two FSV free online software, the first one was 
developed by the Aquila University in Italy (Orlandi, 2006). The second one has been 
developed by the Electromagnetic Compatibility Group of the Universitat Politécnica de 

Catalunya (GCEM, 2011). GCEM-UPC has built and developed new tools for the traditional 

FSV allowing the user to evaluate the graphics in a very quick and easy way (for more 

information visit: http://www.upc.edu/web/gcem/ files /FSV.exe). 

The FSV method is based on the decomposition of the results into two groups; the first one 

discusses the difference in amplitude (Amplitude Difference Measure, ADM) and the 

second one the difference between the characteristic of the signals (Feature Difference 

Measure, FDM). The combination of these two indicators (ADM and FDM) is a 

measurement of the overall difference (Global Difference Measure, GDM) (A. P. Duffy et al., 

2006; Orlandi et al., 2006). 

All indicators ADM, FDM and GDM have the ability to be configured to perform a point-to- 

point analysis. The advantage of relying on a point-to-point data is to know which areas of 

the data sets have the major differences. A subscript "i" is added to consider this point-by- 

point feature (ADMi, FDMi and GDM i). 

Another way to qualitatively analyse the FSV indicators is represented by a probability 

density function. It is useful for a rapid and comprehensive analysis of the results. This 

indicator uses a histogram that can be divided into six categories: excellent, very good, 
good, fair, poor, very poor. 

Finally, a technique that has proved useful in presenting and interpreting FSV data, 

particularly the confidence histograms, is a “Grade and Spread” (G/S) diagram 

(Archambeault, A. P. Duffy, & Orlandi, 2009; Archambeault & Yu, 2009).The Spread serves a 

similar purpose to variance or standard deviation in statistical methods and is a 

measurement of the spread of a distribution. The Grade is a measurement of the quality of 

the results and serves a similar purpose to skew measurements in statistics. It is important 
to remember that Grade and Spread must be used together, since if only one is used, the 
interpretation can be inaccurate. 

The FSV method requires a serie of steps to obtain each indicator, a brief summary of some 

of them are following described: 

a. The first step is to interpolate the two sets of data to be compared to having the same 
number of samples for comparison. 

b. Once both datasets have the same number of samples, the Fourier Transformer is 
applied. Then, a high pass filter is applied in the dataset obtained the "Hi" data. The 
same procedure with a band pass filter is done to obtain the "Lo" data. An important 
aspect to consider is that these two new dataset (Hi and Lo) are separated by the 
breaking point, which is chosen with 40% of all data. 

c. Knowing all values of "Lo" data, the ADM indicator is calculated according to: 
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d. The next step calculated is the FDM. It is composed of three parts based on the 
derivatives calculated in the last step. The numerical values in the equations are parts of 
the heuristic and have been determined empirically. 
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FDMo) = 2(|FDM1 + FDMaqj) + FDM3(|) (25) 
e. The GDMi indicator is calculated using the ADM and FDM indicators, as shown in 


equation (26). 
GDMiqy = [ADM?, + FDM?) (26) 


f. Calculation of the mean value (XDMtot). After the ADM, FDM and GDM point-by- 
point values are calculated, it is possible to find the average value (27). These indicators 
are very useful to evaluate the quality of the results with one number. 


n r 
XDMtot = di=1 XDMIQ) (27) 
n 


g. Calculation of the confidence histogram. This is the term that is most often used in the 
descriptions of the quality of comparisons. The determination of the histogram (Fig 1.) 
is simply a case of counting the proportion of points that fall into one of the categories, 
according to the rule base in Table 1. 
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XDMc value (X=A,F,G) 

XDMc<x0.1 Excellent 
0.1<XDMcs0.2 Very Good 
0.2<XDMc<0.4 Good 
0.4<XDMc<0.8 Fair 
0.8<XDMcs1.6 Poor 

1.6<XDMc Very Poor 


Table 1. XDMc interpretation scale. 


ADMc 
0.6 


0.5 


0.4 


0.3 
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0.0 


VG G F VP 
Fig. 1. Confidence histogram example (ADMc). 


It is quite possible that in some cases there is significant spread between the different 
categories (Excellent (Ex), Very Good (VG), Good (G), Fair (F), Poor (P), and Very Poor 
(VP)). Often this spread is caused when the datasets are very noisy. A quick and easy 
solution is to use a wider window width which produces a greater degree of smoothing. 

The "Grade" and the "Spread" are calculated based on the ADMc and FDMc. The "Grade" is 
calculated by taking the number of category, starting from the best (Excellent) to the worst 
(very poor), which include a user defined amount (named “threshold”, an 85 % is 
recommended) of total samples of data sets to being compared. The “Spread” is similar to a 
typical standard deviation since it also determines how many categories are required to 
include 85% of the data, but the starting category is the highest rated (instead of the 
excellent category, as in the Grade). 

Despite all the benefits offered by the FSV, in the validation processes, some studies found 
problems when trying to use it to analyze transient signals (R Jauregui, Riu, & Silva, 2010; Ri 
Jauregui, Silva, Orlandi, Sasse, & A. Duffy, 2010). In particular, the main problem was in the 
ADM indicator. 

The problem with the ADM indicator lies in the way it calculates the dataset "Lo" (R 
Jauregui, Rojas-Mora, & Silva, 2011). This dataset is obtained through the breakpoint (IBP), 
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which is calculated with the 40% of the signal, because it assumes that most of the signal 
energy content is within this range. The problem is that many times in a transient signal up 
to 90% of the energy can be contained in the first peaks. Therefore, it is highly probable that 
only the first peak of the transient is considered, and the other low level differences are not 
taken into account when comparing with FSV method. For this reason, before using this 
method to validate a new type of signal that has not been previously tested, it is recommend 
making a small review and analyzing the consistency of results. 


3.5 Validation Transient Signals in Time Domain (VTTD) 
One of the most interesting signals from the viewpoint of the numerical simulation is the 
impulsive noise, also known as transient phenomenon. These types of signals are used in the 
time domain for analyzing large frequency bands. On the other hand, these same signals 
offer a challenge in areas such as electromagnetism or resistance of structures in building. 
The transient signals could be described as a signal that varies between two consecutive 
steady states during a short period of time compared to the time scale of interest. In other 
words, there must be a "momentary" change of the magnitude seen for a very short time in 
the sense that this short interval of time should be much less than one cycle of the signal. 
Thanks to the different derivatives performed, it only takes into account the changing 
intervals in the graphs without giving attention to the level differences affected. This 
particular feature makes it difficult to analyze the effect using common methods of 
validation (R Jauregui, 2009). For this reason, a special method was developed to perform 
the Validation of the Transient in Time Domain (VTTD). It is proposed to use five indicators 
to assess the different parameters of the transient data sets: 

a. Feature Difference Measure (FSV-FDM). The calculation of this indicator is made 

using the equations of the FDM specified in the FSV method (see equation (25)). Unlike 
the amplitude indicator (ADM), the FDM does not present any problem when it is used 
to analyse the transient in time domain. Thanks to the different derivatives performed 
in this indicator, it only takes into account the changing intervals in the graphs without 
attention to the level differences. 
This indicator is applied before taking into account any other indicators, because its 
value determines whether or not to continue the validation process. Some studies (R 
Jauregui, Pous, Fernandez, & Silva, 2010; R Jauregui, Silva, & Riu, 2007) determined that 
the optimal limit for a correct interpretation is when the FDMror indicator is equal or 
less than 0.8. This value ensures that the two data sets (numerical simulation and 
measurement, for example) have a similarity that is within the acceptable margin. 

b. Amplitude Pulse Level (APL). This indicator measures the difference between the 
maximum amplitude of the signals. The maximum level of a transient signal is very 
important because it can produce several types of problems. Thus, the APL indicator 
aims to assess the maximum amplitude level difference between the two data sets. 
According to the equations (28) the APL calculates the difference of the maximum of 
each data set in absolute value to guarantee that the analysis is independent of the 
polarity. 

|max(¥1) — max(Y2)| 


= ea AA, 28 
APL = maD] |max(Y¥1)| > |max(Y2)| (28) 
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|max(¥2) — max(Y1)| 2 
APL = — eo |max(Y¥1)| < |max(Y2)| (29) 
Where max (Y1) is the maximum magnitude for the first dataset and max (Y2) is the 
corresponding one for the second dataset. An APL range result is from 0 to 1. When the 
APL is equal to 0 the similarity is perfect, but as they increase, the result moves to 1. 

c. Maximum Rise Time (MRT). One important issue in a transient signal is the rise time. 
The lower it is, the more contents of the disturbance are on the high-frequency band, 
which is usually a problem in validation analysis. The calculation of this indicator is 
very similar to the one used in APL; the only difference is that it calculates the first 
derivative (30) and then applies the equations (31) or (32). 


Jao aJ i 
p(t ena (30) 
xi — x) i = {1,2,3 ..n} 
pe) DY2 
irs ON wa a e aa (31) 


Imax(D*1)| 


Y2 Y1 
opr = eet?) L < |max(D¥?)| (32) 
|max(D*)| 

Where i is the number of the point (from 1 to n). j is the set of the graph that we want to 
analyze (Y1 is the first one and Y2 the second one). Dii is the derivative for each point (1 
to n) for both dataset. |max(D¥1)| is the absolute maximum value of the derivative of 
the first dataset and |max(DY2)| is the derivative of the second one. Similar to the APL 
indicator, the equation (31) or (32) is applied in order to ensure that MTR varies from 0 
to 1. 

d. Energy Contained in the Signals (ECS). This indicator measures the energy contained 
in the transients. In many cases, a transient energy could be very significantly affecting 
the behaviour of a system; therefore, it is important to evaluate. Applying the equations 
(34) or (35), the difference of energy between both datasets can be determined for the 
same interval of time. 


tn 5 
: = {Y1, Y2} 

s= | UÍ (t) dt AST (33) 
N © ty <tn 
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ECS = ~p [je | |E”? (34) 
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ECS = -rA |EYt] < |E”?] (35) 


Where “E” represent the energy and “U” is the magnitude recorded for each dataset, 
both datasets must be defined from t = 0 to t = tn. 

e. The Total Error Average (TEA). As seen in previous validation methods, it is very 
useful to have an indicator that reflects the overall quality of results. This indicator 
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allows a quick and simple way to have a general idea of the quality of results. The 
indicator TEA meets this objective quickly and easily. 

The calculation of TEA is based on finding the squared error of the indicators FDM, 
APL, MRT and ECS as shown in equation (36). In this equation, a weighting factor for 


each indicator can be defined by “a”, “p”, “y” to highlight the importance of a 
particular indicator in a particular situation. 


TEAS [SEDER ORT + EE) 036) 
at+B+y 


The intervals of this indicator may vary depending on the transient and the type of 
problem being analyzed. It is the user who must define the scales and values that define 
it. For example, in the case of Electromagnetic Compatibility (EMC) area, a useful scale 
to analyze the transients is: 
Good: from 0 to 0.3 
Regular: from 0.3 to 0.5. 
Bad: from 0.5 to 1. 
This method allows rapid and objective quantification of the simulation results, but it is 
important to note that this validation method is valid only to study the transient in time 
domain. 


4. Validation method application examples 


In order to show the application of the different validation methods previously presented, 
two real cases are chosen to compare their results. Before making any comparison, it is 
necessary to normalize all validation methods to ensure that all of them are within the same 
scale (from 0 to 1). As it is usual, we use different categories to help us to identify the quality 
of the results: excellent (from 0 to 0.16), very good (from 0.17 to 0.34), good (from 0.35 to 0.5), 
fair (from 0.51 to 0.65), poor (from 0.66 to 0.80), very poor (from 0.81 to 1). Finally, a survey 
among some experts has been done to compare their opinion with the different validation 
methods under test. 

The VITD method was not used in these examples, as it is defined only for transient 
analysis in time domain. If you need more information about the use or implementation of 
this method, it is recommended to view documents (R Jauregui, 2009; R Jauregui, Riu, & 
Silva, 2010; Riu, R Jauregui, Silva, & Fernandez, 2007). 


4.1 First case of study 

The aim of this first example is to examine the efficiency of each validation method to 
analyze the similarity between two signals. These signals were obtained by the 
measurement (Fig. 2-blue) and the simulation (Fig. 2-red) and show the transfer function 
between two electromagnetically short monopoles inside a resonant cavity. As one can 
observe, the simulation and the measurement have a similar behaviour for the entire 
frequency range. However, some minor differences can be found in the negative resonances. 
In general, the results were classified by a panel of experts with 0.25, which is in the “Very 
good” category. 
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Fig. 2. Comparison between measurement and simulation. 
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Fig. 3. Results from the different validation methods used and the experts opinion. 
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Applying the different validation methods studied before, we obtain the results shown in 
the Fig 3. It is possible to observe that each method gives different results that lead to 
different interpretations. In the cases of the Pearson correlation (0.024) and the Rzj-Factor 
(0.013) methods, the results show that there is a almost a perfect match between the 
measurement and the simulation, and this does not agree with the expert's opinion. 

The main reason for Pearson and Rz;-Factor results is that the correlation method is unable 
to evaluate the differences caused by rapid changes in slopes. This method only analyzes the 
correspondence in amplitude over the signals but no other feature is considered. Because of 
that reason, those methods are not suitable to analyze signals with abrupt changes or, in 
particular, with noise. 

If the focus is now on the third method (Rpenary-Factor), the results show a poor similarity 
between the simulation and measurement. Again, this result does not match with the 
expert’s opinion. The main limitation of this method is that it is very sensitive to sudden 
changes in the signals and the indicators are directly affected. 

Finally, we have the methods of Van Hoven factor, FSV and IELF which results are very 
close to expert’s opinion. This result is not surprisingly, since these three methods are 
particularly robust and have been tested against different types of behaviours. Therefore, as 
it has been explained in the preceding paragraphs, they are ideal for the numerical 
simulations output validation process. 


4.2 Second case of study 

The aim of this second example is to use the validation methods in a more realistic 
application for the field of numerical electromagnetic simulations. In this case, we compare 
a measurement (Fig. 4-Blue) with two different simulations (Fig. 4-red & black). Each 
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Fig. 4. Comparison between measurement and simulations. 
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simulation was performed using the same Finite Difference in Time Domain (FDTD) 
algorithm but with different settings like meshing and time step. The target is to decide, by 
using IELF and FSV methods, which is the simulation that better fits the measurement from 
the point of view of a panel of experts in this topic. 

At a first sight by doing a quick comparison between the signals, anyone can deduce that 
both simulations can be improved, but more difficult is to decide which one is better. When 
we want to study the influence of a particular simulation parameter such as time, mesh, etc. 
it is very important to identify which simulation has a greater similarity with the 
measurement of the real setup which it is supposed to be the right one. 

Some people can realize that first simulation (Fig. 4-red) has a greater similarity than the 
second (Fig. 4-red); since it seems to have a close behaviour with the measurement at high- 
frequencies. However, the simulation number one has important differences at the low- 
frequency band that should not be forgotten because for another set of people this could be 
an important feature where to focus the comparison. Furthermore, the second simulation 
seems to have the opposite behaviour to the first one: a closer similarity at lower 
frequencies, but a significant amplitude difference at higher frequencies. Therefore, in this 
case, it is not an easy task to take an overall decision without the help of an experts group or 
an appropriate validation method. 

Fig 5 shows the results when the IELF and FSV are applied to compare the simulations 1 
and 2 with the measurement. Observing all the indicators of the used methods, one can see 
how both methods are quite close to the experts. These results show that the worst 
simulation is the second one (black) or in other words, the first simulation has more 
similarity with the measurement considering whole plot. 
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Fig. 5. Results of different validation methods used and the expert panel opinion. 
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Even though the two validation methods have a unified approach on which simulation is 
better, each one produces a different value for the final indicator. This is a very common 
problem when several types of validation methods are compared. It is therefore very 
important to always use one single method throughout one validation process. 

Another point of vital importance, when a validation method is chosen, is to have tools that 
help us to analyze the different features present in the signals. This is one of the main 
limitations of the method IELF, because it concentrates all the comparison information in a 
single number. With this method, it is not possible to obtain more information about the 
validation process. 

The FSV method has, as noted above, several analysis tools that can help to establish a more 
comprehensive comparison in the validation process. With the mean value of each indicator 
(Table 2), one can see that the largest difference between the simulations is in the shape 
indicator (FDMror) and not in the amplitude (ADMror) as one might think. 


FSV indicators 


Simulation Í 


FDM tor 
GDMror 


Table 2. FSV mean values indicators. 


One way to analyze in greater detail what happens in each comparison is using the point- 
by-point indicators. It is important to recall that, in this case, the indicators ADMi and FDMi 
correspond to the point-by-point analysis between each simulation and the measurement. 
Fig. 6 shows that the indicator most affected in these comparison is the FDMi indicator and 
of course for the first simulation. Now we can see, very clearly, that the problem is mainly in 
the shape and not in amplitude. 
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Fig. 6. Point-by-point indicators results. (a) ADMi indicator results for simulation 1 & 2 with 
respect to the measurement. (b) FDMi indicator results for simulation 1 & 2 with respect to 
the measurement. 
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Another powerful use of these indicators is to identify where, and the exact value, the major 
differences over all the data set (frequencies in this case) is produced. For both analysis 
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made in this example, it is clear that the major difference is found at frequencies of 0.6, 1.2 

and 1.8 GHz. In the worst case, we can see (Fig. 6b) that the FDMi indicator reaches values 

near to 8, showing an important difference between the first simulation and the 
measurement. 

With these two short examples, it became clear that it is very important to choose a proper 

method of validation that objectively represents the opinion of experts. We have also seen 

how important it is to have the necessary tools to interpret these results. 

It should be noted that all the techniques presented can be used not only to validate the 

numerical methods and simulation; its use can be extended to other areas that require a 

quantitative comparison of complex data. The only important thing when a validation 

method is chosen is that it must provide a similar result to the expert opinion, which implies 
an objective analysis of the data. 

On the other hand, it is needed to take into account that a perfect method to validate any 

kind of result does not exist. Each one of the methods presents advantages and 

disadvantages depending on the type of data and the type of analysis desired. The most 
essential thing at the time to apply the validation is to consider the following items: 

a. The implementation of the validation technique should be as simple as possible; this 
will avoid confusion and data clouding. 

b. The validation method should reflect human opinions. Any technique which leads to 
conflict with the views of the user will fall rapidly into disuse. 

c. The validation method should provide the possibility to be applied in different 
environments and/or applications. 

d. The validation method should be commutative. The results of the comparison should 
always be the same regardless of which is used as a reference or pattern. In other 
words, the user satisfaction and credibility of the method can be affected if the quality 
of the technique varies depending on which data is used as a pattern for comparison. 

e. The validation method must analyse the difference between the two data sets and 
always yield the same result, regardless of the user and number of times the 
comparison is made. 
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